Image processing apparatus and method

ABSTRACT

An image processing apparatus adapted to perform tone correction of luminance of an image, comprises: a image dividing unit configured to divide the image into a plurality of blocks; a calculation unit configured to calculate a plurality of feature amounts of each of the blocks including a luminance value and calculate a saturated feature amount based on the plurality of calculated feature amounts, wherein the saturated feature amount represents color deviation tendency when the tone correction is applied to the image using a reference tone correction characteristic; a changing unit configured to change the reference tone correction characteristic according to the calculated saturated feature amount; and a correction unit configured to perform the tone correction based on the changed reference tone correction, wherein the changing unit weakens a degree of the reference tone correction characteristic when the saturated feature amount indicates a stronger tendency toward color deviation.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus andmethod, and more particularly, to an image processing apparatus and amethod which performs tone correction on inputted images.

2. Description of the Related Art

Conventional techniques for performing tone correction of images involvemaking shadow corrections to correct luminance of dark portionscontained in the images, making contrast corrections to increasecontrast of the images, and so on. For example, a large number of darkportions are contained in an image which has a luminance histogram suchas shown in FIG. 10A, and if the luminance of the dark portions isincreased using a correction curve such as shown in FIG. 10B, an imageof more appropriate brightness can be obtained. In many photographyscenes, such tone correction allows photographed images to be correctedinto more desirable images.

However, when such a correction is applied to a scene whose imagecontains large high-chroma regions, the balance among color componentsof pixels in the high-chroma regions can be lost, resulting in colordeviation. Suppose, for example, FIG. 10A is a luminance histogram of ascene in which a vivid red flower is shown in the center of an image andan 8-bit RGB value in part of the flower is RGB=(255, 56, 94). Theluminance obtained from a conversion formula defined by sRGB and givenby Eq. (1) below is Y=approximately 120.Y=0.299R+0.587G+0.114B  (1)

The calculated luminance of 120 is plotted at a point indicated by x inthe luminance histogram of FIG. 10A, and it can be seen that theluminance is not very high. The luminance undergoes a substantialcorrection as can be seen from the amount of correction for x in FIG.10B. However, in order to apply shadow correction to such a region whilemaintaining RGB color balance, R needs to exceed 255. However, if oneattempts to express the R, G, and B values in 8 bits, an R component iscropped at 255. As a result, only G and B components are raised with Rremaining the same as before. Consequently, the RGB color balancechanges from the original ratio, causing color deviation.

To deal with this problem, a technique disclosed in Japanese PatentLaid-Open No. 2007-243542 performs chroma correction by calculatingchroma in a central portion of an image and determining an amount ofcorrection based on a proportion of high-chroma pixels in the region andthe like.

Also, a technique disclosed in Japanese Patent Laid-Open No. 2002-335536calculates amounts of correction on a pixel by pixel basis at theluminance of the respective pixels, and then calculates actual amountsof correction by reducing the calculated amounts of correctiondifferently according to the chroma of the pixels, where the higher thechroma, the greater the reduction.

However, when a technique, such as the technique disclosed in JapanesePatent Laid-Open No. 2007-243542, which is based solely on theproportion in which high-chroma pixels are contained in a given region,is applied to tone corrections, the amount of correction could besuppressed with respect to scenes to which tone correction can normallybe applied safely. For example, in shadow correction, the amounts ofcorrection are generally reduced with respect to regions whose luminanceis too low or regions whose luminance is too high as shown in FIG. 10B,in order to give a more natural look. Even when tone correction such asshown in FIG. 10B is applied to a scene which contains a large number ofhigh-chroma regions, if the regions have high luminance or lowluminance, color deviation is less likely to occur. The correction suchas shown in FIG. 10B may result in a good image corrected to anappropriate brightness as a whole. However, if tone correction isweakened simply for the reason that the scene contains a large number ofhigh-chroma regions, tone corrections which are safe to make will alsobe suppressed.

On the other hand, a technique, such as the technique disclosed inJapanese Patent Laid-Open No. 2002-335536, which checks chroma anddetermines the amounts of correction on a pixel by pixel basis, needs tocalculate information for determination of high chroma, including a hueand chroma, on a pixel by pixel basis, and then calculate an appropriateamount of correction at the chroma level of each pixel. Consequently,calculation time increases with increases in pixel count. Furthermore,if tone correction is applied on a pixel by pixel basis to a scene suchas a flower whose chroma is not uniform and varies with the shade,although the amounts of correction of the regions whose chroma issaturated are suppressed, the surrounding regions whose chroma is notsaturated are subjected to correction. Although this enables makingcorrections without causing chromatic saturation, chromatic details ofthe original image could be lost. As an example, FIG. 10C showsvariations in the chroma of a linear region in a scene showing a vividred flower. When corrections are made to the scene on a pixel by pixelbasis, the variations in chroma become as shown in FIG. 10D. When FIG.10C and FIG. 10D are compared, fine variations in chroma have been lostin FIG. 10D after the corrections. In particular, when chromatic detailsare reduced in high-chroma regions, one could feel as though there werecolor deviations even though actually there is no color deviation.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and is designed to perform tone correction suitable for ahigh-chroma scene which is liable to undergo color deviation as a resultof correction.

According to the present invention, provided is an image processingapparatus adapted to perform tone correction of luminance of an image,comprising: an image dividing unit configured to divide the image into aplurality of blocks; a calculation unit configured to calculate aplurality of feature amounts of each of the blocks including a luminancevalue and calculate a saturated feature amount based on the plurality ofcalculated feature amounts, wherein the saturated feature amountrepresents color deviation tendency when the tone correction is appliedto the image using a reference tone correction characteristic; achanging unit configured to change the reference tone correctioncharacteristic according to the calculated saturated feature amount; anda correction unit configured to perform the tone correction based on thereference tone correction characteristic changed by the changing unit,wherein the changing unit weakens a degree of the reference tonecorrection characteristic of the tone correction when the saturatedfeature amount indicates a stronger tendency toward color deviation.

According to the present invention, provided is an image processingmethod for performing tone correction of luminance of an image,comprising: a block dividing step of dividing the image into a pluralityof blocks; a calculation step of calculating a plurality of featureamounts of each of the blocks including a luminance value andcalculating a saturated feature amount based on the plurality ofcalculated feature amounts, wherein the saturated feature amountrepresents color deviation tendency when the tone correction is appliedto the image using a reference tone correction characteristic; achanging step of changing the reference tone correction characteristicaccording to the calculated saturated feature amount; and a correctionstep of performing the tone correction based on the reference tonecorrection characteristic changed by the changing step, wherein thechanging step weakens a degree of the reference tone correctioncharacteristic of the tone correction when the saturated feature amountindicates a stronger tendency toward color deviation.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing a schematic configuration of an imagesensing apparatus according to an embodiment of the present invention;

FIGS. 2A and 2B are flowcharts showing tone correction processesaccording to first, third, and fourth embodiments;

FIGS. 3A to 3D are diagrams showing an example of weights used incalculating saturated feature amounts according to the first embodiment;

FIGS. 4A to 4D are diagrams showing an example of how to weaken amountsof correction according to the first embodiment;

FIGS. 5A and 5B are diagrams showing an example of how to weaken amountsof correction in a scene with a face and a scene with no face accordingto a second embodiment;

FIGS. 6A and 6B are flowcharts showing tone correction processesaccording to the second embodiment;

FIGS. 7A to 7C are diagrams showing an example of a scene containing anextremely low-luminance region according to the third embodiment;

FIGS. 8A to 8C are diagrams showing an example of how to select blocksaccording to the fourth embodiment;

FIG. 9 is a block configuration diagram showing an informationprocessing apparatus according to an embodiment; and

FIGS. 10A to 10D are diagrams illustrating conventional problems withhigh-chroma scenes.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described indetail in accordance with the accompanying drawings.

First Embodiment

In a first embodiment, description will be given of a method forcalculating a saturated feature amount which corresponds to a colordeviation tendency of a given scene of a photographic image anddetermining an amount of correction for actual use according to thesaturated feature amount.

FIG. 1 is a block diagram showing a configuration of an image sensingapparatus 100 as an example of an image processing apparatus accordingto the first embodiment. The image sensing apparatus 100 can be adigital camera, digital video camera, or other apparatuses which canobtain image data by imaging a subject. In FIG. 1, an optical system 101is made up of a lens, shutter, and diaphragm and is adapted to focus anappropriate amount of light from a subject onto an image sensing device102 with an appropriate timing. The image sensing device 102 convertsthe light focused through the optical system 101 into image data. A CPU103 performs various computations and controls various parts of theimage sensing apparatus 100 according to inputted signals and a program.

A primary storage device 104 is adapted to store temporary data and usedas a workspace for the CPU 103. A secondary storage device 105 stores aprogram (firmware) and various settings information used to control theimage sensing apparatus 100. A storage medium 106 stores photographicimage data and the like. The storage medium 106 is able to be removedafter photo-taking and inserted into a personal computer or the like toread data. That is, the image sensing apparatus 100 may be of any typeas long as it has capabilities to access the storage medium 106 and readand write data from/to the storage medium 106. A display unit 107 isadapted to display a viewfinder image during photo-taking, displayphotographed images, display characters for interactive operation, andso on. A console unit 108 is used to accept user actions. Buttons,levers, a touch panel, and the like can be used for the console unit108.

A communications device 109 is adapted to connect to external devicesand exchange control commands and data therewith. PTP (Picture TransferProtocol) is used as a protocol in establishing a connection andconducting data communications. The communications device 109 mayconduct communications via a wired connection such as a USB (UniversalSerial Bus) cable or via a wireless connection such as a wireless LAN.Also, the communications device 109 may connect to the external devicesdirectly, via a server, or via a network such as the Internet.

According to the first embodiment, FIG. 2A is a diagram showing a flowof correcting luminance tone for an image. With reference to FIG. 2A, adescription will be given below of an example of calculating a saturatedfeature amount which corresponds to a color deviation tendency of aphotographic image and weakening an amount of correction for actual useaccording to the saturated feature amount.

First, an image is shot through the optical system 101 and image sensingdevice 102 (step S201). Then, the CPU 103 divides the obtained imageinto plural blocks (step S202). Incidentally, according to the firstembodiment, the number of divided blocks has been prescribed in advance,but may be changed according to the photography scene or subject. Also,an image stored in the storage medium 106 may be read out and handled inthis step.

Next, the CPU 103 calculates a luminance feature amount of each of theblocks obtained in step S202 (step S203). According to the firstembodiment, the CPU 103 acquires average values of 8-bit RGB signals ofthe pixels contained in each block, calculates an 8-bit luminance signalY using above-described Eq. (1) as the luminance feature amount of theblock. Incidentally, the luminance feature amount does not always needto be calculated from the average values of RGB signals. For example, aconceivable method involves calculating respective signal values(boundary values) Mr, Mg, and Mb for R, G, and B signals of the pixelsin the block such that the number of pixels whose R, G, and B signalvalues are equal to or larger than the respective signal values Mr, Mg,and Mb will respectively make up N % of all the pixels in the block, andthen calculating the luminance signal Y from the values of Mr, Mg, andMb using Eq. (1). Also, although in the first embodiment, the luminancesignal Y is calculated using Eq. (1), this is not restrictive, and the Gsignal may be used directly in a simplified manner or the luminancefeature amount may be obtained by adding other information.

Once the luminance feature amount of each block is obtained in stepS203, the CPU 103 further calculates a color feature amount of eachblock (step S204). According to the first embodiment, the CPU 103calculates an 8-bit chroma signal S and brightness signal V from averagevalues R, G, B of the RGB signals of each block obtained in step S203,using Eq. (2) below, and designates the chroma signal S and brightnesssignal V as a color feature amount of the block.

$\begin{matrix}{S = \left\{ {{\begin{matrix}{0,} & {V = 0} \\{\frac{{\max\left( {R,G,B} \right)} - {\min\left( {R,G,B} \right)}}{\max\left( {R,G,B} \right)},} & {V > 0}\end{matrix}V} = {\max\left( {R,G,B} \right)}} \right.} & (2)\end{matrix}$where max(R, G, B) is the largest of R, G, and B values and min(R, G, B)is the smallest of R, G, and B values.

As with the luminance feature amount, the color feature amount does notalways need to be calculated from the average values of RGB signals. Forexample, a conceivable method involves calculating respective signalvalues (boundary values) Mr, Mg, and Mb for R, G, and B signals of thepixels in the block such that the number of pixels whose R, G, and Bsignal values are equal to or larger than the respective signal valuesMr, Mg, and Mb will respectively make up N % of all the pixels in theblock, and then calculating the chroma signal S and brightness signal Vfrom the values of Mr, Mg, and Mb using Eq. (2). Incidentally, althoughaccording to the first embodiment, the chroma signal S and brightnesssignal V are calculated using Eq. (2), the color feature amount may bedefined by one of the two signals, or by adding other information.

After the luminance feature amounts and color feature amounts arecalculated on a block by block basis in steps S203 and S204, the CPU 103calculates saturated feature amounts (step S205). The saturated featureamount represents the color deviation tendency resulting from tonecorrection, thus the higher the degree to which each block satisfies theconditions described below, the higher the saturated feature amount isset to be.

(1) The luminance is neither too high nor too low.

(2) The difference between a maximum value and minimum value of a signalcomponent (e.g., RGB) of a pixel is large.

(3) Any of signal components of a pixel is close to a maximum value (255in the case of an 8-bit signal).

First, whether or not condition (1) is satisfied can be determined basedon the luminance Y calculated in step S203. As described above, theamount of tone correction is generally reduced in the case of regionshaving too low luminance or too high luminance. Thus, when a blocksatisfies condition (1), this means that the block is susceptible totone correction and that the signal components in the block will changegreatly as a result of the tone correction.

Next, whether or not condition (2) is satisfied can be determined basedon the chroma S calculated in step S204. Depending on the tonecorrection technique, differences among signal components of the pixelmay be small. Thus, when a block satisfies condition (2), this meansthat the difference between a maximum signal component and minimumsignal component in the block could become small after correction evenif the difference is large before the correction.

Also, whether or not condition (3) is satisfied can be determined basedon the brightness V calculated in step S204. When tone correction isapplied in such a way as to increase brightness, the signal componentsof the pixel subjected to the correction are increased in value.However, when condition (3) is satisfied, this means that the increasedvalue of the signal component could exceed the maximum value availableto the signal component, resulting in color deviation.

According to the first embodiment, in order to determine whether or notconditions (1) to (3) are satisfied, luminance weights, chroma weights,and brightness weights are calculated based on the luminance featureamount and color feature amount obtained in steps S203 and S204, and acombination of all thereof is used as a saturated feature amount of thegiven block. The weights are calculated using lines, such as shown inFIGS. 3A to 3C, prepared in advance.

FIG. 3A, which corresponds to condition (1), shows a line graph for useto calculate luminance weights according to the luminance featureamount. The line graph is made up of a polygonal line peaking at apredetermined luminance value Y₀. The predetermined luminance value Y₀has its value varied based on a reference correction line (correctioncharacteristics) to detect accurately whether or not a correction willcause color deviation. For example, when the reference correction lineis used to increase the size and brightness of a dark portion, the valueof Y₀ is set close to the dark portion and when the reference correctionline is used to increase the size and brightness of a light portion, thevalue of Y₀ is set close to the light portion.

FIG. 3B, which corresponds to condition (2), shows a line graph for useto calculate chroma weights according to the value of chroma S which isa color feature amount. When a predetermined chromatic value S₀ isexceeded, the weight is increased to ensure that the saturated featureamount will take a value which represents a stronger tendency towardcolor deviation.

FIG. 3C shows a line graph for use to calculate brightness weightsaccording to the value of brightness V which is a color feature amount.When a predetermined brightness value V₀ is exceeded, the weight isincreased to ensure that the saturated feature amount will take a valuewhich represents a stronger tendency toward color deviation.

The combination of all the weighted feature amounts are designated asthe saturated feature amount of the given block. Incidentally, althoughin the first embodiment, the maximum values of the weights for the threetypes of feature amount are set to the same value as shown in FIGS. 3Ato 3C, this is not always necessary. For example, the maximum value ofweights for the luminance feature amount may be set to a smaller valuethan the maximum value of the weights for the other feature amounts toreduce the influence of luminance.

In step S205, the CPU 103 calculates the saturated feature amounts on ablock by block basis using Eq. (3), and further calculates the saturatedfeature amount of the entire image of the scene. According to the firstembodiment, the saturated feature amount C of the entire image iscalculated from the saturated feature amounts c_(i) calculated on ablock by block basis, using Eq. (3) below.

$\begin{matrix}{{c_{i} = {{{wY}\left( Y_{i} \right)} \times {{wS}\left( S_{i} \right)} \times {{wV}\left( V_{i} \right)}}}{C = \frac{c_{1} + c_{2} + \ldots + c_{N}}{N}}} & (3)\end{matrix}$where Y_(i), S_(i), and V_(i) are the luminance value, chromatic value,and brightness value of the ith block, respectively; N is the number ofblocks; and wY(Y_(i)), wS(S_(i)), and wV(V_(i)) are output weightsobtained, respectively, from input values Y_(i), S_(i), and V_(i) basedon FIGS. 3A to 3C.

Although the saturated feature amounts are calculated in Eq. (3) bytreating all the blocks equivalently, different weights may be assignedto different areas to carry out calculations based on Eq. (4) below byincreasing weights W(i) assigned to blocks nearer to the center, forexample, as shown in FIG. 3D.

$\begin{matrix}{{c_{i} = {{{wT}\left( Y_{i} \right)} \times {{wS}\left( S_{i} \right)} \times {{wV}\left( V_{i} \right)} \times {W(i)}}}{C = \frac{c_{1} + c_{2} + \ldots + c_{N}}{\sum\limits_{i = 1}^{N}{W(i)}}}} & (4)\end{matrix}$

Once the saturated feature amount of the scene is calculated in stepS205, the CPU 103 generates a correction curve for tone correction ofthe scene. First, the CPU 103 calculates correction feature amounts foruse to determine reference correction characteristics (tone correctioncharacteristics) (step S206). In calculating the correction featureamounts, the CPU 103 establishes a curve such as shown in FIG. 4A. Ifthe average value of a histogram obtained as a correction feature amountis Ain, the amount of correction is Avol. When the average value of thehistogram exceeds a predetermined value as shown in FIG. 4A, the largerthe average value, the smaller the amount of correction is set to be.Then, based on the calculated correction feature amounts, the CPU 103calculates a reference tone correction amount suitable for the scene(step S207). The reference tone correction amount can be calculatedusing the average value of the histogram obtained in step S206 as input,based on the relationship shown in FIG. 4A. Normally, output luminanceAout (=Ain+Avol) is calculated using the correction feature amountsobtained in step S206 as input luminance Ain, based on the referencetone correction amount obtained in step S207. Next, by setting a controlpoint at an intersection of Ain and Aout as shown in FIG. 4C, the CPU103 acquires a correction curve for tone correction such as indicated bya broken line in FIG. 4C. Such a correction curve is useful in making adesirable correction, provided the scene does not contain a high-chromaregion. However, if the scene contains high-chroma regions, the use of acorrection curve such as indicated by the broken line in FIG. 4C couldcause color deviation in the high-chroma regions. Thus, the correctioncurve is varied based on the saturated feature amount obtained in stepS205.

To vary the correction curve, first in step S208, the CPU 103 weakensthe reference tone correction amount calculated in step S207, accordingto the saturated feature amount. For example, by defining a relationshipbetween the saturated feature amount and a ratio of weakening L as shownin FIG. 4B, the ratio of weakening L to be applied to the amount ofcorrection of the scene can be calculated from the calculated saturatedfeature amount. The output luminance Aout is changed from the inputluminance Ain based on Eq. (5) below, using the ratio of weakening L.Aout′=Ain+(Aout−Ain)×(1−L)  (5)

This makes it possible to calculate a correction curve which can addressa scene which is likely to undergo color deviation, by establishingcontrol points at points Ain and Aout′ as shown in FIG. 4C (step S209).To prevent darkening of dark portions and highlight-detail loss of lightportions, control points may be added also to the dark portions andlight portions.

Also, the correction curve may be found before placing restrictionsusing the saturated feature amount. In that case, steps S210 and S211are carried out as shown in FIG. 2B instead of steps S208 and S209. Thismakes it possible to calculate the correction curve from control pointsas indicated above by a broken line in FIG. 4D, then weaken the amountof correction of the entire correction curve, and thereby calculate acorrection curve suitable for a scene which is likely to undergo colordeviation. For example, if a correction curve Yout=f(Yin) is obtained instep S210, it is possible to calculate a correction curve Yout′=f(Yin)suitable for a scene which is likely to undergo color deviation based onthe ratio of weakening L, using Eq. (6) below.Yout′=Yin +(Yout−Yin)×(1−L)  (6)

Next, a luminance tone correction is made using the changed correctioncurve.

As described above, according to the first embodiment, luminance is usedas a criterion in calculating the saturated feature amount of a scene.This makes it possible to reduce color deviation of high-chroma sceneprone to color deviation when corrections are made and make appropriatecorrections to a scene less prone to color deviation in spite of highchroma when corrections are made.

Furthermore, by dividing an image into blocks and processing the imageon a block by block basis, it is possible to check whether or not thereis a region which as a whole is likely to undergo color deviation ratherthan whether or not local pixels are likely to undergo color deviation.This enables making more desirable corrections to a scene containingchromatic variations.

Second Embodiment

An example of weakening the amount of correction when there are a largenumber of regions which are likely to undergo color deviation has beendescribed in the first embodiment, but a desirable ratio of weakeningvaries with the photography scene. Thus, in the second embodiment, theway of weakening the amount of correction is varied with the photographyscene.

Examples of differences in the photography scene include the presence orabsence of a face. Let us consider a case in which a correction curvefor use to correct luminance Ain to luminance Aout has been calculatedas shown by a broken line in FIG. 5B. FIG. 5B is an enlarged view of asection around Ain in the correction curve of FIG. 4C. If the screencontains a wide high-chroma region, but no face, human attention tendsto be focused on the high-chroma region. Therefore, if correction ofsuch a scene can result in color deviation of the high-chroma region, itis highly likely that the correction will go well if importance is givento reduction of color deviation. Thus, in such a case, the higher thelikelihood of color deviation, the more the tone correction is weakened.The control point is brought closer to a point C in FIG. 5B to make theless correction to be applied to the high-chroma region.

However, if there is a face as well as a wide high-chroma region whichis likely to undergo color deviation, human attention tends to be drawnto both the high-chroma region and the face. Therefore, in the casewhere the facial region is originally dark, even though there is a wideregion which is likely to undergo color deviation, if tone correction isweakened too much, an image undesirable in terms of the face willresult. That is, in the case of a scene with a face, if the controlpoint is brought close to the point C in FIG. 5B as in the case of ascene without a face, it will become impossible to correct the dark faceeven if the two scenes are similar.

To deal with this, a control point B different from the point C is setin advance in a scene with a face. The higher the likelihood of thescene to undergo color deviation, the closer the control point isbrought to the point B, to apply necessary correction to some extentwhile reducing color deviation. Thus, in the second embodiment, as shownin FIGS. 6A and 6B, the process of recognizing a subject (step S301) isadded to the flowcharts according to the first embodiment described inFIGS. 2A and 2B, and a correction feature amount calculation process(step S306) and correction amount calculation process (step S307)different from the first embodiment are performed. Other processes arethe same as the first embodiment described above, and thus the samereference numerals as in FIGS. 2A and 2B are used and description willbe omitted as appropriate.

According to the second embodiment, the CPU 103 detects a subject in theimage obtained in step S201 (step S301). Next, in steps S202 to S205,the CPU 103 calculates the saturated feature amounts in the mannerdescribed in the first embodiment.

Next, the CPU 103 calculates the correction feature amount (step S306).According to the second embodiment, if, for example, a face is detectedas a subject in step S301, the luminance of the face is calculated asthe correction feature amount. If no face is detected, the CPU 103determines the correction feature amount in the same manner as in stepS206 of FIGS. 2A and 2B. For example, if the average value of ahistogram obtained as a correction feature amount is Ain, the amount ofcorrection of the average value Ain becomes Avol. Consequently, byestablishing a curve such as shown in FIG. 5A, the amount of correctioncan be decreased gradually with increases in the average value of thehistogram over a predetermined value. Once the correction feature amountis calculated, the CPU 103 calculates a reference tone correction amountaccording to the luminance of the face, using the relationship shown inFIG. 5A (step S307).

Once the reference tone correction amount is calculated in step S307,the CPU 103 determines the ratio of weakening L and generates acorrection curve in steps S208 and S209 or in steps S210 and S211. Instep S209 or S211, the ratio of weakening L is varied depending onwhether or not a specific subject has been detected in step S301.

For example, whereas in the first embodiment, the output luminance Aoutis changed from the input luminance Ain based on Eq. (5), in the secondembodiment, the output luminance Aout is changed from the inputluminance Ain based on Eq. (7) below.Aout′=Aout, Aout≦AlimAout′=Alim+(Aout−Alim)×(1−L), Aout>Alim  (7)where Alim is a minimum guaranteed tone correction amount and set to ahigher value than Ain.

Consequently, when a scene has a very high likelihood of color deviationand contains no face, if the control point is moved from the point A tothe point C as shown in FIG. 5B, the amount of correction can berestricted. When a similar scene contains a face, if the control pointis moved from the point A to the point B as shown in FIG. 5B, theoverall amount of correction can be restricted while making somecorrections to the face.

As described above, the second embodiment can calculate a more desirableamount of correction with respect to a scene containing a specificsubject such as a face which can draw the user's attention.

Note that although a human face is detected as a specific subject in thesecond embodiment described above, the present invention is not limitedto this and may be applied to any subject as long as the subject isspecified in advance.

Third Embodiment

The proportion of a region which is likely to undergo color deviation inthe entire screen is checked in the first embodiment, but if, forexample, extremely dark/light regions are contained in the screen, humanattention tends to be drawn to a region of appropriate brightnessexcluding the extremely dark/light regions. For example, FIG. 7A shows ascene in which the region other than a high-chroma region is extremelydark. If the region other than a high-chroma region have someappropriate luminance as in the case of a normal landscape rather thanhaving extremely low luminance such as shown in FIG. 7A and thehigh-chroma region makes up a small proportion of the entire screen,human attention is less prone to be drawn to the high-chroma region. Insuch a case, if the luminance of the region other than the high-chromaregion is only slightly dark, corrections made to increase brightness ofthe dark region tend to produce a more desirable image. However, in FIG.7A, the region other than the high-chroma region has extremely lowluminance and has a histogram such as that shown in FIG. 7B. In such acase, human attention tends to be drawn to the high-chroma region inFIG. 7A. In such a scene, even though the high-chroma region makes up asmall proportion of the entire screen, if the ratio of weakening ofcorrection is reduced as in the case of normal scenes, color deviationcould occur in the high-chroma region which tends to draw attention.

Thus according to the third embodiment, a plurality of luminancethresholds are set, as indicated by broken lines in FIG. 7B. Next, incalculation of the luminance feature amount and color feature amount,pixels within the thresholds are used as valid pixels to calculate thefeature amount of the block, and an amount of correction more suitablefor the scene is calculated.

A flow of basic processes according to the third embodiment is similarto the flow according to the first embodiment described with referenceto FIGS. 2A and 2B, but operations of a luminance feature amountcalculation process in step S203 and color feature amount calculationprocess in step S204 differ from the first embodiment. Whereas in thefirst embodiment, the luminance feature amount and color feature amountare calculated from the R, G, and B averages of all the pixels in theblock, in the third embodiment, luminance thresholds are set and R, G,and B average values are calculated using only the pixels within athreshold range. As the thresholds are set for luminance, a parameterused in determining the R, G, and B averages varies with the extent towhich extremely dark/light regions are contained in the block.Consequently, reliability of each block varies with the effective pixelcount which represents the number of pixels whose luminance is withinthe threshold range in the block. Thus, if the weight W(i) in Eq. (4)described above is used as the effective pixel count of the block, thesaturated feature amount C can be calculated by taking the reliabilityinto consideration. This process can be applied to the scene shown inFIG. 7A as follows.

First, saturated feature amounts c_(i) are determined on a block byblock basis. As a result, the saturated feature amounts c_(i) oflow-luminance blocks are close to 0 and the saturated feature amountsc_(i) of high-chroma regions are close to 1, as shown in FIG. 7C. Then,the saturated feature amount of the entire scene is determined using Eq.(3) as in the case of the first embodiment, and the result is as shownby Eq. (8) below.

$\begin{matrix}{C = {\frac{0.0 + 0.0 + \ldots + 1.0 + \ldots + 0.0}{N}\frac{1}{16}}} & (8)\end{matrix}$

In this way, although most regions in the screen have extremely lowluminance and a high-chroma region tends to draw attention, actually thesaturated feature amount C of the scene is estimated to be low. Thus, inthe third embodiment, the effective pixel count within the effectiveluminance range described above is taken into consideration. Forexample, in a scene such as shown in FIG. 7A, the effective pixel countof the region having extremely low luminance is set close to 0 as withe_(i) values shown in FIG. 7C. With this in view, when e_(i) is used asthe weight W(i) in Eq. (4), the saturated feature amount C of the entirescene becomes as shown by Eq. (9) below.

$\begin{matrix}{C = {\frac{{0.0*0.0} + {0.0*0.0} + \ldots + {1.0*0.9} + \ldots + {0.0*0.0}}{0.0 + 0.0 + \ldots + 0.9 + \ldots + 0.0} = 1.0}} & (9)\end{matrix}$

In this way, since a high-chroma region can be estimated to have a highsaturated feature amount of a scene which tends to draw attention, it ispossible to calculate an ratio of weakening appropriate for ahigh-chroma region.

As described above, the third embodiment can calculate a more desirabletone correction amount with respect to a scene containing a large numberof extremely dark/light regions.

Fourth Embodiment

In a fourth embodiment, an example of a method for dividing an imageinto blocks will be described. Regarding the size of the blocks, thelarger the size, the easier it is to recognize regions which are likelyto undergo color deviation, as an area in the image. Therefore, divisioninto large-size blocks makes it easy to recognize and detect regionswhich are likely to undergo color deviation, for example, when a largepart of a block is occupied by a specific subject such as a high-chromabuilding or car or in the case of macro-photography of a flower.However, when a block contains different subjects, such as when a blockcontains both flowers and leaves as in a distance shot of a high-chromaplant, the subjects are averaged, and consequently, the chroma isestimated to be low.

Let us take as an example a scene shown in FIGS. 8A to 8C. In FIG. 8A,the scene is divided into large-size blocks and two of the blocks areextracted. In this case, the block on the left side is almost entirelyoccupied by a flower region. Consequently, even if there are somevariations in chroma within the range, the region which is likely toundergo color deviation can be recognized as an area. However, the blockon the right side contains both flowers and leaves, which consequentlyare averaged, making it impossible to extract the block as a regionwhich is likely to undergo color deviation.

In FIG. 8B, the scene is divided into small-size blocks and 18 of theblocks are extracted. In this case, each of the nine blocks on the rightside corresponds to either a flower region or leaf region. This makes itpossible to extract regions which are likely to undergo color deviationand regions which are not likely to undergo color deviation. Incontrast, the nine blocks on the left side cause the large flower to belooked at closely. Consequently, if the flower has shades, influence offine chromatic variations comes into play, making it impossible to seewhether or not the flower is likely to undergo color deviation as awhole. The fourth embodiment allows a more desirable tone correctionamount to be calculated even in such a case.

A flow of basic processes according to the fourth embodiment is similarto the flow according to the first embodiment described with referenceto FIGS. 2A and 2B, but the fourth embodiment differs from the firstembodiment in that an image is divided into blocks of different sizes.In the first embodiment described above, in calculating the luminancefeature amounts and color feature amounts of blocks in steps S203 andS204, R, G, and B average values are calculated on a block by blockbasis and luminance values, chromatic values, and brightness values arecalculated therefrom. On the other hand, in the fourth embodiment, twodifferent block sizes are defined in advance. Regarding the larger blocksize, average values AveRL, AveGL, and AveBL of the R values, G values,and B values of the pixels contained in the blocks as well as variancesVarR, VarG, and VarB of the R values, G values, and B values of thepixels contained in the blocks are determined in advance. Regarding thesmaller block size, average values AveRS, AveGS, and AveBS of the Rvalues, G values, and B values of the pixels contained in the blocks arecalculated in advance.

That is, the average value and variance of each block in FIG. 8A and theaverage value of each block in FIG. 8B are calculated. Next, variancesof the larger blocks are checked. The variances are low when a blockcontains only a flower as in the case of the block on the left side ofFIG. 8A. Conversely, the variances have high values when a blockcontains different subjects such as flowers and leaves as in the case ofthe block on the right side of FIG. 8A. That is, in such a case,preferably smaller-size blocks are used for determination instead of thelarger-size block. Thus, when the variances calculated from alarger-size block are larger than a predetermined threshold, the averagevalues calculated from smaller-size blocks are used. On the other hand,when the variances calculated from a larger-size block are equal to orsmaller than the threshold, the average values calculated from thelarger-size block are used as they are. Incidentally, in thethresholding, determinations can also be made based on, for example,whether or not the average value or maximum value of VarR, VarG, andVarB is larger than a certain threshold Th. Consequently, as shown inFIG. 8C, the average value calculated from the larger-size block can beused when only a flower region is contained in the block and the averagevalues calculated from smaller-size blocks can be used when a flowerregion and leaf region are contained in the block.

In so doing, if Eq. (4) is used to calculate the saturated featureamounts and the pixel count of each block is used as the weight W(i),the saturated feature amounts can be calculated by taking the blocksizes into consideration.

Incidentally, although two block sizes are used for division in thefourth embodiment, three or more block sizes may be used alternatively.

While exemplary embodiments of the present invention have been describedabove, it is to be understood that the present invention is not limitedto the disclosed exemplary embodiments and that various modificationsand changes can be made within the scope of the invention.

Other Embodiments

It should be noted that the present invention may be applied to a systemmade up of two or more devices (e.g., a host computer, interface device,camera head, scanner, and the like) as well as to an apparatus (e.g., adigital still camera, digital video camera, or the like) made up of asingle device.

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., computer-readable medium).

FIG. 9 is a block diagram showing a schematic configuration of aninformation processing apparatus 200 capable of implementing the presentinvention using the program described above. In FIG. 9, a display unit201 displays images before image processing, images after imageprocessing, characters for interactive operation, and the like. Aconsole unit 202 is used to accept user actions. A keyboard, pointingdevice, and the like can be used for the console unit 202. A CPU 203performs various computations and controls respective parts of theinformation processing apparatus 200 according to inputted signals and aprogram. A primary storage device 204 is adapted to store temporary dataand used as a workspace for the CPU 203. A secondary storage device 205stores a program and various settings information used to perform imageprocessing. A communications device 206 connects to external devices andexchanges control commands and data therewith. A form similar to thecommunications device 109 may be selected for the communications device206.

When the present invention is implemented by the information processingapparatus 200, images stored in the secondary storage device 205 orimages in general acquired via the communications device 206 may beprocessed, and the images which can be processed are not limited tophotographic images.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2011-023248, filed on Feb. 4, 2011 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus adapted to performtone correction of luminance of an image, comprising: an image dividingunit configured to divide the image into a plurality of blocks; acalculation unit configured to calculate a plurality of feature amountsof each of the blocks including a luminance value and calculate asaturated feature amount based on the plurality of calculated featureamounts, wherein the saturated feature amount represents color deviationtendency when the tone correction is applied to the image using areference tone correction characteristic; a changing unit configured tochange the reference tone correction characteristic according to thecalculated saturated feature amount; and a correction unit configured toperform the tone correction based on the reference tone correctioncharacteristic changed by said changing unit, wherein said changing unitweakens a degree of the reference tone correction characteristic of thetone correction when the saturated feature amount indicates a strongertendency toward color deviation.
 2. The image processing apparatusaccording to claim 1, wherein the plurality of feature amounts include achromatic value.
 3. The image processing apparatus according to claim 2,wherein the saturated feature amount takes a value which represents astronger tendency toward color deviation when any of the chromaticvalues of the respective blocks is larger than a predetermined chromaticvalue.
 4. The image processing apparatus according to claim 1, whereinthe plurality of feature amounts include a brightness value.
 5. Theimage processing apparatus according to claim 4, wherein the saturatedfeature amount takes a value which represents a stronger tendency towardcolor deviation when any of the brightness values of the respectiveblocks is larger than a predetermined brightness value.
 6. The imageprocessing apparatus according to claim 1, further comprising adetection unit configured to detect a predetermined subject in theimage, wherein said changing unit makes a change to weaken degree of thetone correction to a lesser extent when the subject is detected thanwhen the subject is not detected.
 7. The image processing apparatusaccording to claim 1, wherein said calculation unit calculates thefeature amounts of each of the blocks using signal values of pixelswhose luminance values are within a predetermined range.
 8. The imageprocessing apparatus according to claim 1, wherein: said image dividingunit divides the image into blocks of a plurality of different sizes bydetermining variances of larger blocks, and dividing the blocks whosevariances are equal to or smaller than a threshold into larger blockswhile dividing the blocks whose variances are larger than the thresholdinto smaller blocks; and said calculation unit calculates the saturatedfeature amount by assigning weights corresponding to sizes of the blocksto the plurality of feature amounts calculated for each of the blocks.9. The image processing apparatus according to claim 1, wherein saidcalculation unit calculates the plurality of feature amounts usingaverage values of signal components of the pixels contained in therespective blocks.
 10. The image processing apparatus according to claim1, wherein said calculation unit calculates the saturated feature amountfrom the plurality of feature amounts using a boundary value calculatedsuch that more than a predetermined number of signal components of thepixels which exceed the boundary value will be contained in each of theblocks.
 11. An image processing method for performing tone correction ofluminance of an image in an image processing apparatus, comprising: ablock dividing step of dividing the image into a plurality of blocks; acalculation step of calculating a plurality of feature amounts of eachof the blocks including a luminance value and calculating a saturatedfeature amount based on the plurality of calculated feature amounts,wherein the saturated feature amount represents color deviation tendencywhen the tone correction is applied to the image using a reference tonecorrection characteristic; a changing step of changing the referencetone correction characteristic according to the calculated saturatedfeature amount; and a correction step of performing the tone correctionbased on the reference tone correction characteristic changed by saidchanging step, wherein said changing step weakens a degree of thereference tone correction characteristic of the tone correction when thesaturated feature amount indicates a stronger tendency toward colordeviation.
 12. A non-transitory readable storage medium having storedthereon a program which is executable by an image processing apparatus,the program having a program code for realizing the image processingmethod according to claim
 11. 13. An image processing apparatus adaptedto perform tone correction of luminance of an image, comprising: animage dividing unit configured to divide the image into a plurality ofblocks; a calculation unit configured to calculate a luminance value anda chromatic value of each of the blocks; a correction unit configured toperform the tone correction based on the luminance values and thechromatic values, wherein said correction unit performs a weaker tonecorrection as a number of blocks having the chromatic values which arelarger than a predetermined value increases.
 14. The image processingapparatus according to claim 13, wherein said calculation unitcalculates a brightness value of each of the blocks, and said correctionunit performs a weaker tone correction as a number of blocks having thebrightness values which are larger than a predetermined value increases.15. The image processing apparatus according to claim 13, furthercomprising a detection unit configured to detect a predetermined subjectin the image, wherein said correction unit performs a weaker tonecorrection in a case where the subject is not detected than in a casewhere the subject is detected.
 16. The image processing apparatusaccording to claim 13, wherein said calculation unit calculates thechromatic value of each of the blocks using signal values of pixelswhose luminance values are within a predetermined range.
 17. The imageprocessing apparatus according to claim 13, wherein said calculationunit calculates a saturated feature amount by calculating a weight ofeach of the blocks corresponding to the luminance value and thechromatic value of each of the blocks, and said correction unit performsthe tone correction based on the saturated feature amount.
 18. An imageprocessing apparatus adapted to perform tone correction of luminance ofan image, comprising: an image dividing unit configured to divide theimage into a plurality of blocks; a calculation unit configured tocalculate a luminance value and a brightness value of each of theblocks; a correction unit configured to perform the tone correctionbased on the luminance values and the brightness values, wherein saidcorrection unit performs a weaker tone correction as a number of blockshaving the brightness values which are larger than a predetermined valueincreases.
 19. An image processing method for performing tone correctionof luminance of an image in an image processing apparatus, comprising:an image dividing step of dividing the image into a plurality of blocks;a calculation step of calculating a luminance value and a chromaticvalue of each of the blocks; a correction step of performing the tonecorrection based on the luminance values and the chromatic values,wherein said correction step performs a weaker tone correction as anumber of blocks having the chromatic values which are larger than apredetermined value increases.
 20. An image processing method forperforming tone correction of luminance of an image in an imageprocessing apparatus, comprising: an image dividing step of dividing theimage into a plurality of blocks; a calculation step of calculating aluminance value and a brightness value of each of the blocks; acorrection step of performing the tone correction based on the luminancevalues and the brightness values, wherein said correction step performsa weaker tone correction as a number of blocks having the brightnessvalues which are larger than a predetermined value increases.